Multi-Source Knowledge Reasoning Graph Network for Multi-Modal Commonsense Inference

As a crucial part of natural language processing, event-centered commonsense inference task has attracted increasing attention. With given observed event, the intention and reaction people involved in event are required to be inferred with artificial intelligent algorithms. To solve this problem, sequence-to-sequence methods widely studied, where is first encoded into specific representation then decoded generate results. However, all existing learn only textual information, while visual information ignored, which actually helpful for reference. In article, we define new multi-modal reference both information. A dataset also provided. Then propose multi-source knowledge reasoning graph network task, three kinds relational considered. Multi-modal correlations learned get event’s from global perspective. Intra-event object relations explored capture fine-grained feature an graph. Inter-event semantic through external understand associations among events We conduct extensive experiments on dataset, results show effectiveness our method.